Effects of Rank and Precision of Search Results on Users’ Evaluations of System Performance

Previous research has demonstrated that system performance does not always correlate positively with user performance, and that users often assign positive evaluation scores to systems even when they are unable to complete tasks successfully. This paper investigates the relationship between actual system performance and users’ perceptions of system performance by manipulating the level of performance experienced by users and measuring users’ evaluations of system performance. Eighty-one subjects participated in one of three laboratory studies. The first two studies investigated the impact of the location (or rank order) of five relevant and five non-relevant documents in a search results list containing ten results. The third study investigated the impact of varying levels of precision (.30, .40, .50 and .60) of a search results list containing ten results. Results demonstrate statistically significant relationships between precision and subjects’ evaluations of system performance, and ranking and subjects’ evaluations of system performance. Of the two, precision explained more variance in subjects’ evaluation ratings and was a stronger predictor of subjects’ ratings. Finally, the number of documents subjects examined significantly influenced their evaluations, even when the difference was a single document.